URL-Enhanced Adaptive Page-Refresh Models

We study the refresh model required to keep an up to date copy of a web page. This has applications for more efficient and frequent crawling, in the case of search engines, and for higher hit rates for proxy servers with pre-fetching. Two main models have been proposed namely the uniform refresh model (Cho and Garcia-Molina, 2000) and the adaptive page refresh model (Edwards et al., 2001), with some debate as to the relative value of each model. In this work we show that adaptive page refresh models can be further improved by careful selection of initial page-refresh rates of newly added links as indicated by our page evolution studies showing that page-change rates (and consequently page-refresh rates) are dependent on top-level domain, category and page depth.